docs: #22 scalability measured at 100,000 and 1,000,000 albums #73

erikhfj · 2026-11-04T19:28:04Z

Stacked on #72. This branch is based on feat/21-postgres-tests-in-ci because it adds the integration CI job that the other benchmarks are read against, and because it is the PR whose CI run proves the new job works. Merge #72 first; the diff here then shrinks to the three benchmark scripts and the README section.

Closes #22.

What was done

The README already had 16 October numbers from a single 100,000-album run with no plans and no index comparison, which is what the issue asked for. This replaces them with measurements at 100,000 and 1,000,000 albums, each with the plan PostgreSQL chose, and adds the two comparisons the issue asks for.

Three committed, read-only scripts make it reproducible:

server/scripts/benchmark.sh - runs all checks, refuses any database whose name does not end in _perf (the index comparison takes an exclusive lock on album; the live crate database must not be reachable from it). Seeding stays a separate, destructive step so no benchmark runner can wipe a catalog by accident.
server/scripts/paging-perf.sql - cursor page vs OFFSET at a configurable depth (-v offset=500000).
server/scripts/index-perf.sql - the same selective search with and without the GIN index. DROP INDEX runs inside a transaction that is rolled back, so the index is restored even if the script fails.

What was measured

Method: PostgreSQL 18.3, laptop, warm cache, median of five runs (SQL) and three (paging/HTTP). Synthetic data only, in crate_100k_perf and crate_perf, never crate. Seeding 10 s / 95 s; 41 MB / 350 MB, GIN 11 MB / 75 MB.

Operation	100k	1M
Selective full-text search (Bitmap Index Scan)	2.6 ms	11.7 ms
Broad full-text search, matches every row (Parallel Seq Scan)	16.3 ms	83.7 ms
Two genres AND + two decades	20.9 ms	87.6 ms
Full broad `searchAlbums` request with facets and counts	~100 ms	~580 ms

Depth	100k cursor	100k OFFSET	1M cursor	1M OFFSET
50,000	26 ms	219 ms	295 ms	865 ms
500,000	-	-	227 ms	1167 ms

Findings I did not paper over

The broad query does not use the GIN index, and should not. It matches every album, so reading most of the index and then most of the table costs more than one sequential scan. The README now says this instead of implying the index is always used.
The GIN index is worth 5-6x on a selective search (11.7 vs 62.3 ms at 1M), not the 10x+ a "index makes search fast" story would suggest, because the synthetic artist name repeats. Documented as measured.
Cursor paging is depth-independent, but not free. Keyset is faster than OFFSET because it discards before sorting, not because it avoids the scan: there is no index on the sort key, so both still scan and sort. With a composite (lower(title), id) index the cursor page drops to 0.09 ms at 500,000 rows while OFFSET needs 164 ms - that 47 MB index is the missing piece, and it is not added here. It needs its own write-cost and bærekraft measurement, so I am filing it as a follow-up issue rather than slipping a migration into a documentation PR.

What was tested

Every number comes from the committed scripts, run against fresh databases. benchmark.sh was run end to end, and deliberately against a database called crate to check the guard:

::error::Refusing to run: 'crate' is not a *_perf database.

pnpm format:check && pnpm lint passes. The HTTP rows use server/scripts/search-albums-perf.json against a production build of server/dist/index.js.

The numbers are laptop measurements, not VM measurements. The README says so; I am not claiming VM numbers I did not take.

The database-backed suites were skipped in CI because nothing set TEST_DATABASE_URL, so "27 skipped" was passing for the whole database layer. The new `integration` job creates a throwaway PostgreSQL cluster inside the runner's temp directory, socket-only, applies the migrations, runs the server suite and deletes the cluster. No shared test database, no secret to hand out, and the application's `crate` database is never in reach. `deploy` now requires the job, so a deploy cannot ignore the database tests. The cluster alone would still have left the failure mode from "skipped" to "silently empty", so `server/scripts/assert-postgres-tests.mjs` reads the JSON report and fails when any database-backed file has zero executed or any pending assertions. Verified against both reports: with the database it prints the four files and exits 0, without it fails on the three suites that skip.

The first CI run failed with 'initdb: command not found' in 12 seconds: Debian puts the server binaries in /usr/lib/postgresql/<major>/bin and only the client on PATH, so PostgreSQL 18 being installed is not enough. Try PATH first, then the newest installed major version, and fail with a message that names the missing package if neither works.

Documents the "udefinert stor datasamling" requirement with numbers instead of an assertion, and records the limits honestly. New: `paging-perf.sql` compares the keyset page with OFFSET at a configurable depth; `index-perf.sql` runs the selective search with and without the GIN index, dropping it inside a transaction that is rolled back so it is always restored; `benchmark.sh` runs both and refuses any database whose name does not end in `_perf`, because the index comparison takes an exclusive lock on album. Findings worth keeping: the broad full-text query is planned as a sequential scan even with the index, and that is correct; the GIN index is worth 5-6x on a selective search; cursor paging is depth-independent while OFFSET reaches 1.2 s at 500,000 rows, but only becomes O(page) with a composite sort index, which the schema does not have yet. That is measured, explained and filed as a follow-up rather than added in a documentation issue.

rachelks

Nice work on the scalability measurements! The results are useful, and I like that you've documented the limitations as well. I found a few things worth looking into before merging:

Paging benchmark: The benchmark uses a tuple comparison, while the API uses an OR condition. Even though these are logically equivalent, PostgreSQL may produce different execution plans, so the benchmark might not accurately represent the API's performance. The cursor and OFFSET queries also seem to be off by one, meaning they don't return exactly the same page. Could we align the queries with the API and make sure the comparison is consistent?
Database safety: The benchmark script checks the database name in the URL, but connection parameters can override it. Could we also verify the actual connected database using current_database()? This would make the safeguard more reliable.
Reproduction instructions: The README sets DATABASE_URL, but the psql commands don't explicitly use it. The HTTP benchmark instructions could also clarify how to start the API against the performance database, so others can reproduce the measurements.
Indexed paging results: The README includes results from an experiment with an additional composite index, but I couldn't find instructions for reproducing it. Could we include the index creation command and a representative query plan?

One small documentation suggestion as well: the new scalability section contains a lot of useful information, but I found it a bit difficult to identify the main results and conclusions among all the technical details. Maybe we could keep the key findings and a short explanation in the README, and move the more detailed experiments and measurements into a separate document under docs/, ex. scalability.md? That would make the README easier to navigate while keeping the technical evidence available.

Overall, the measurements and explanations are valuable. I think these adjustments would make the benchmarks easier to verify and the documentation easier to follow.

Two problems with paging-perf.sql, both from review by rachelks on #73. The cursor and OFFSET pages were one row apart. The anchor row was fetched with OFFSET :offset LIMIT 1, which is the row after the last one already seen, so the keyset query returned rows offset+2..offset+21 while the OFFSET query returned offset+1..offset+20. The anchor is now OFFSET :offset - 1, so both start at the same row. The script now proves it instead of asserting it: it materialises both pages and reports cursor_rows, offset_rows and shared_rows. A run at 500,000 rows prints 20 / 20 / 20, so the timings compare the same work. The keyset predicate is documented as the row comparison `(a, b) > (x, y)` that the API uses, with the reason it matters: PostgreSQL applies it as an index condition on a matching composite index, while the equivalent OR form is only a filter. That OR form was in crateForUser when this was written; #77 changed the API to the row comparison, so the benchmark and the API now run the same shape. offset is documented as "rows already scrolled past", which must be at least 1 because the first page has no cursor to page after. The script says so and stops with ON_ERROR_STOP instead of measuring a mislabelled page.

Review by rachelks on #73 asked how the (lower(title), id) numbers were produced, and whether the database name check could be bypassed. paging-index-perf.sql is the missing experiment: it creates the index inside a transaction, runs ANALYZE, measures the keyset page and the OFFSET page at the same depth as paging-perf.sql, and rolls the transaction back so the schema is unchanged. Measured at one million albums: 47 MB, cursor 0.11 ms against 180.09 ms for OFFSET at 500,000 rows, with the plan showing `Index Cond: (ROW(lower(title), id) > ROW(...))` and `Index Searches: 1`. benchmark.sh now runs it, and checks the real connection in addition to the name in the URL. libpq lets `?dbname=`, a service file or PGDATABASE point somewhere else, and a mistake there runs CREATE INDEX or DROP INDEX on the wrong database. It asks the server for current_database() and refuses anything not ending in _perf. Verified with `postgres:///crate_perf?host=...&dbname=crate_test`: the URL check passes, the server check fails, exit 1.

Review by rachelks on #73: the scalability section buried the conclusions in several screens of tables and plans. README keeps the headline table, five one-line conclusions and the pointer. The method, the full tables, the plans, the prefix-search numbers and the reproduction steps move to docs/scalability.md, which is linked from the README and from the scripts. The reproduction section is corrected as part of the move: the psql commands now pass "$DATABASE_URL" explicitly (psql does not read it on its own, which made the old snippet connect to the default database), and it shows how to build and start the API against the performance database for the HTTP numbers.

erikhfj · 2026-11-06T01:20:32Z

Thanks — all four were worth fixing. Pushed as aaefc8a, 14fa014, 4724fd4.

1. The two pages really were one row apart, and the tuple is now the API's form

You are right on both counts.

The anchor was fetched with OFFSET :offset LIMIT 1, which is the row after the last one already scrolled, so the keyset query returned rows offset+2..offset+21 while OFFSET :offset returned offset+1..offset+20. The anchor is now OFFSET :offset - 1 LIMIT 1.

Rather than assert that they match, the script now checks it: it materialises both pages and reports the counts before timing them. At 500,000 rows:

 cursor_rows | offset_rows | shared_rows
-------------+-------------+-------------
          20 |          20 |          20

On the tuple/OR difference: this PR branched before #77, and #77 changes crateForUser from

added_at < $1 OR (added_at = $1 AND id < $2)

to the row comparison

(added_at, id) < ($1, $2)

so the benchmark now matches the API. That was not cosmetic: the OR form is a filter even with a matching composite index, while the row comparison becomes an index condition (plan in point 4). The script and docs/scalability.md now say which form they measure and why. Merge order is #77 before #73, otherwise the document would describe a resolver that is not on main yet.

One correction to the depth parameter while I was in there: offset is "rows already scrolled past", which must be at least 1. The script says so and stops, instead of silently measuring the first page under a deep-page label.

2. Check the connection, not the URL

Added. benchmark.sh still rejects a non-_perf name in the URL before connecting (so a typo cannot reach the app database at all), then asks the server:

connected_db="$(psql "$DATABASE_URL" -X -tAc 'select current_database()')"

Both must end in _perf. Verified with a URL that lies:

DATABASE_URL='postgres:///crate_perf?host=...&dbname=crate_test' server/scripts/benchmark.sh
::error::connected to 'crate_test', which is not a *_perf database.
exit 1

3. psql and the API in the reproduction steps

Correct — psql does not read DATABASE_URL, so the old snippet connected to the default database. The steps now export it once and pass "$DATABASE_URL" to every psql. They also show how to build and start the API against the performance database for the HTTP numbers:

pnpm --filter @t19/server build
PORT=3001 node server/dist/index.js &

4. The indexed experiment is now reproducible

New server/scripts/paging-index-perf.sql: creates the index inside a transaction, ANALYZE, measures, rolls back, so the schema is unchanged either way.

CREATE INDEX album_title_lower_id_perf_idx ON album (lower(title), id);

At one million albums it reports 47 MB and, at 500,000 rows in:

cursor:  Execution Time: 0.110 ms
  ->  Index Scan using album_title_lower_id_perf_idx on album
        Index Cond: (ROW(lower(title), id) > ROW('synthetic album 549998'::text, 549998))
        Index Searches: 1

OFFSET:  Execution Time: 180.093 ms
  ->  Index Scan using album_title_lower_id_perf_idx on album (actual time=0.027..169.267 rows=500020 loops=1)

benchmark.sh runs it too, so "runs all of them" stays true. Building and rolling back the index costs about 3.5 s.

Documentation split

Agreed that the README was hard to navigate. It now keeps the headline table, five short conclusions and a link. docs/scalability.md holds the method, the full tables, the plans, the prefix-search numbers and the reproduction steps, including the ones above. The prefix-search section moved with the rest of the measurements, unchanged.

erikhfj requested a review from a team as a code owner November 4, 2026 19:28

erikhfj added 2 commits November 4, 2026 13:35

erikhfj force-pushed the docs/22-scalability-benchmark branch from 51c80bf to bbeb34d Compare November 4, 2026 19:35

rachelks reviewed Nov 5, 2026

View reviewed changes

erikhfj added 3 commits November 5, 2026 19:16

erikhfj requested a review from rachelks November 6, 2026 01:20

Uh oh!

docs: #22 scalability measured at 100,000 and 1,000,000 albums #73

docs: #22 scalability measured at 100,000 and 1,000,000 albums #73

erikhfj commented Nov 4, 2026 •

edited

Loading

Uh oh!

rachelks left a comment

Uh oh!

erikhfj commented Nov 6, 2026

Uh oh!

Uh oh!

docs: #22 scalability measured at 100,000 and 1,000,000 albums #73

Are you sure you want to change the base?

docs: #22 scalability measured at 100,000 and 1,000,000 albums #73

Conversation

erikhfj commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was done

What was measured

Findings I did not paper over

What was tested

Uh oh!

rachelks left a comment

Choose a reason for hiding this comment

Uh oh!

erikhfj commented Nov 6, 2026

Uh oh!

Uh oh!

erikhfj commented Nov 4, 2026 •

edited

Loading